Skip to content

fix(sglang): force reasoning when the template prefills the think tag - #11790

Open
pos-ei-don wants to merge 1 commit into
mudler:masterfrom
pos-ei-don:sglang-force-reasoning-prefilled-think-tag
Open

fix(sglang): force reasoning when the template prefills the think tag#11790
pos-ei-don wants to merge 1 commit into
mudler:masterfrom
pos-ei-don:sglang-force-reasoning-prefilled-think-tag

Conversation

@pos-ei-don

Copy link
Copy Markdown
Contributor

Description

Qwen3-style chat templates append the opening <think> tag to the prompt when thinking
is enabled. The model therefore never generates it and emits only the reasoning text plus the
closing </think>.

sglang's ReasoningParser keys off the opening tag:

in_reasoning = self._in_reasoning or self.think_start_token in text
if not in_reasoning:
    return StreamingParseResult(normal_text=text)

With such a template the entire completion — reasoning, the raw </think>, and the answer —
comes back as content, and reasoning_content stays empty no matter how reasoning_parser
is configured:

ReasoningParser("qwen3", stream_reasoning=False)                        reasoning    0 chars
ReasoningParser("qwen3", stream_reasoning=False, force_reasoning=True)  reasoning  746 chars
                                                                        content     28 chars

sglang's own OpenAI server solves this with

force_reasoning = (self.template_manager.force_reasoning
                   or self._get_reasoning_from_request(request))

This backend has no template manager, so the same signal is derived from the rendered prompt:
if it ends with the detector's think_start_token, the tag was prefilled and the parser is
constructed with force_reasoning=True.

Notes for Reviewers

Why this is conditional rather than always on. Forcing unconditionally breaks the
thinking-off case — a completion with no tags at all is then filed entirely as reasoning and
the answer disappears:

19 chars of output, no </think>
  without force : reasoning   0    content  19   "2,3,5,7,11,13,17,19"
  with    force : reasoning  19    content   0   ""      <- answer gone

force_reasoning is only passed when it is meant to be True, so detector defaults
(DeepSeek-R1 already defaults to True) are untouched.

Structured decoding is the exception, and it matters. A grammar applies from the first
token, so the model cannot emit the closing tag even though the template opened the block.
The whole completion is schema output and belongs in content; forcing there files it as
reasoning and returns an empty answer. Measured against a JSON-schema code audit:
10107 characters of "reasoning", zero content. sglang's own server keeps the two apart
for the same reason — its grammar backend owns the reasoning prefix when a reasoning parser
is configured. Hence the grammar_constrained guard, with a regression test for it.

The construction is factored into _new_reasoning_parser() so the streaming and
non-streaming paths, which previously built the parser separately, cannot drift apart.

Tested against Qwen3-style models on an sglang backend; backend/python/sglang/test.py covers
the prefilled-tag case, the thinking-off case and the grammar case.

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

Qwen3-style chat templates append the opening <think> tag to the *prompt*
when thinking is enabled. The model therefore never generates it and emits
only the reasoning text plus the closing </think>.

sglang's ReasoningParser keys off the opening tag:

    in_reasoning = self._in_reasoning or self.think_start_token in text
    if not in_reasoning:
        return StreamingParseResult(normal_text=text)

so with such a template the entire completion — reasoning and answer, the
raw </think> in between — is returned as content and reasoning_content
stays empty, no matter how reasoning_parser is configured.

sglang's own OpenAI server handles this via

    force_reasoning = (self.template_manager.force_reasoning
                       or self._get_reasoning_from_request(request))

This backend has no template manager, so derive the same signal from the
rendered prompt: if it ends with the detector's think_start_token, the tag
was prefilled and the parser is constructed with force_reasoning=True.

Structured decoding is the exception, and it matters: a grammar applies
from the first token, so the model cannot emit the closing tag even though
the template opened the block. The whole completion is schema output and
belongs in content — forcing there files it as reasoning and returns an
empty answer. Measured against a JSON-schema code audit: 10107 characters
of "reasoning", zero content. sglang's own server keeps the two apart for
the same reason; its grammar backend owns the reasoning prefix when a
reasoning parser is configured.

force_reasoning is only passed when it is meant to be True, so detector
defaults (DeepSeek-R1 already defaults to True) are untouched, and a
prompt without a prefilled tag behaves exactly as before — which matters,
because forcing unconditionally makes an answer generated with thinking
off disappear into reasoning_content.

The construction is factored into _new_reasoning_parser() so the streaming
and non-streaming paths, which previously built the parser separately,
cannot drift apart.

Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>

@localai-org-maint-bot localai-org-maint-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to merge from my review. The prefilled-think-tag detection is scoped correctly, grammar-constrained output is preserved as content, and both parser paths share the same construction logic. Regression coverage, Python syntax checks, and diff checks look clean. @mudler

@localai-org-maint-bot

Copy link
Copy Markdown
Collaborator

Review pass. The reasoning is correct and matches how sglang's own server handles this via template_manager.force_reasoning.

Why it is right: when Qwen3's template appends <think> to the prompt, the completion only ever contains the closing tag, so a detector keyed on the opening tag files everything as content and reasoning_content comes back empty. Deriving the signal from the rendered prompt is the only option here, since this backend has no template manager.

The subtle part is also right: the grammar_constrained carve-out. With structured decoding the model cannot emit the closing tag, so forcing would file schema output as reasoning. And force_reasoning is only passed when it means True, so detectors with their own default (DeepSeek-R1) are untouched.

Three small things, none blocking:

  • When forcing, the parser is constructed twice, once to read detector.think_start_token and once with force_reasoning=True. Cheap, but the token could be read from the class instead.
  • except TypeError around ReasoningParser(force_reasoning=True, ...) swallows any TypeError, not just an unknown-kwarg one, so a genuine signature error inside the detector degrades silently to the unforced parser.
  • The new specs construct a real sglang ReasoningParser with model_type="qwen3" and depend on the installed sglang supporting the force_reasoning kwarg. If either is missing, _new_reasoning_parser returns None and the test dies on an AttributeError rather than skipping.

That last one matters more than it looks, because backend/python/sglang/test.py does not run in CI (no tests-sglang job; it is commented out in test-extra.yml), so nobody would notice until running it locally.

Conflicts with #11786 on sglang/test.py, so whichever lands first forces a rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants